Apparatus, method and computer program for computer vision

ABSTRACT

An apparatus comprising circuitry configured to transfer motion information obtained from a plurality of sensors of different or similar type to a common representation.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is based on PCT filing PCT/EP2018/064246, filedMay 30, 2018, which claims priority to EP 17174267.9, filed Jun. 2,2017, the entire contents of each are incorporated herein by reference.

TECHNICAL FIELD

The present disclosure generally pertains to the field of computervision, in particular to camera and radar-based measurement systems.Such technology can for example be used in applications for automotivedriving.

TECHNICAL BACKGROUND

In autonomous driving and in the field of advanced driver assistancesystems the surrounding scenery is typically described as a small numberof unextended contacts, each being described by a state with at least 2dposition and 2d velocity that represents motion.

In designing autonomous cars, a number of sensors such as cameras, radarand LiDAR are used to provide the information necessary for self-drivingwithout direct contact. For example, modern self-driving cars rely ontechniques such as stereo cameras, Doppler radar and/or LiDAR (LightDetection and Ranging) to estimate motion.

Stereo cameras provide high resolution information and they can seecolor which makes them very efficient in the classification of objectsand texture interpretation. Stereo cameras can also be efficiently usedfor motion estimation. Estimating the dense non-rigid three-dimensionalmotion field of a scene is known as scene flow estimation. Scene flowestimation (in short “scene flow”) shows the three-dimensionaldisplacement vector of each surface point between two frames. However,scene flow estimation typically provides motion information with lessaccuracy in the radial direction than in the angular direction.

Doppler radar is a sensor system that uses radio waves to determine thevelocity, range and angle of objects. The radar technology is veryefficient in motion measurement. Radar velocity measurements providegood accuracy in the radial direction but less or even no accuracy inthe angular direction.

Camera and radar based measurement systems thus provide information onmotion in an environment with complementary precision and in verydifferent data representations.

There are already existing technologies and methods for motion measuringof distant objects without direct contact, but all of them sufferuncertainties and they lack comparability between each other.

SUMMARY

According to a first aspect, the disclosure provides an apparatuscomprising circuitry configured to transfer motion information obtainedfrom a plurality of sensors of different or similar type to a commonrepresentation.

According to a further aspect, the disclosure provides a methodcomprising transferring motion information obtained from a plurality ofsensors of different or similar type to a common representation.

According to a yet further aspect the disclosure provides a computerprogram comprising instructions which, when carried out on a processor,cause the processor to transfer motion information obtained from aplurality of sensors of different or similar type to a commonrepresentation.

Further aspects are set forth in the dependent claims, the followingdescription and the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments are explained by way of example with respect to theaccompanying drawings, in which:

FIG. 1 schematically describes the generation of a fused motion map byfusing data of a Doppler radar and data obtained from scene flowestimation based on images of a stereo camera;

FIG. 2 shows in more detail the alignment of the processed data from theDoppler radar in order to generate a motion map;

FIG. 3 shows in more detail the alignment of the processed data from thestereo camera in order to generate a motion map;

FIG. 4 shows in more detail the process of fusing the motion mapobtained from data of a Doppler radar and the motion map obtained fromdata of a stereo camera in order to obtain a fused motion map;

FIG. 5a schematically shows the uncertainty of a single velocitymeasurement as seen by a scene flow analysis (stereo camera);

FIG. 5b schematically shows the uncertainty of a single velocitymeasurement as seen by a Doppler radar;

FIG. 5c shows the reduced velocity measurement uncertainty in a fusedmotion map generated by fusing the aligned data of a Doppler radar and astereo camera;

FIG. 6 shows an example of a motion map derived from Doppler radarmeasurements;

FIG. 7 shows an example of a motion map derived from scene flowmeasurements based on images captured by a stereo camera;

FIG. 8 shows an example of a fused motion map that combines the motionmap derived from Doppler radar measurements with the motion map derivedfrom scene flow measurements based on images captured by a stereocamera;

FIG. 9 is a block diagram depicting an example of schematicconfiguration of a vehicle control system; and

FIG. 10 is a diagram of assistance in explaining an example ofinstallation positions of an outside-vehicle information detectingsection and an imaging section.

DETAILED DESCRIPTION OF EMBODIMENTS

Before a detailed description of the embodiments under reference of FIG.1, some general explanations are made.

The embodiments described below disclose an apparatus comprisingcircuitry configured to transfer motion information obtained from aplurality of sensors of different or similar type to a commonrepresentation. The plurality of sensors may be of different type or ofsimilar type.

The apparatus may for example comprise an integrated control unit thatcontrols general operation within a vehicle control system in accordancewith various kinds of programs. An integrated control unit may include amicrocomputer, a general-purpose communication I/F, a dedicatedcommunication, a positioning section, a beacon receiving section, anin-vehicle device, a sound/image output section, a vehicle-mountednetwork I/F, a storage section, or the like.

The apparatus may for example be used in a vehicle control system, in anadvanced driver assistance system, in a controller for autonomousdriving, or the like. For example, the apparatus may be applied as avehicle control system or advanced driver assistance system in motorvehicle such a car, a motor cycle, a truck, or the like. Alternatively,the apparatus may also be applied as a vehicle control system oradvanced driver assistance system in an electric vehicle or in a hybridvehicle. The apparatus may also be applied in other scenarios, e.g. ascontroller for autonomous robots or drone products, or, in general, forany movable apparatus. In particular, in the case that the disclosedtechnology is applied to automotive systems, the obtained increase inthe accuracy of the position and velocity information in measurements ofobjects is highly required.

Circuitry may include a processor, a memory (RAM, ROM or the like), astorage, input means (mouse, keyboard, camera, etc.), output means(display (e.g. liquid crystal, (organic) light emitting diode, etc.),loudspeakers, etc., a (wireless) interface, etc.), as it is generallyknown for electronic devices (computers, smartphones, etc.). Moreover,it may include sensors for sensing still image or video image data(image sensor, camera sensor, video sensor, etc.), for sensing afingerprint, for sensing environmental parameters (e.g. radar, humidity,light, temperature), etc.

Sensors of different types may for example be of anyone of the followingtypes: a stereo camera, an ultrasonic sensor, a radar device, or a LiDARdevice (Light detection and Ranging device, or Laser imaging detectionand ranging device).

The circuitry may also be configured to transfer motion informationobtained from a plurality of sensors of similar type to a commonrepresentation. For example, the circuitry may be configured to fuseinformation from several cameras or radars of similar type complementingeach other, e.g. due to different viewing angles.

The common representation to which motion information obtained from theplurality of sensors is transferred may for example be a motion map. Amotion map may also be seen as a “dynamic occupancy map”. A motion mapmay for example be a 2d data array that represents data in the “bird'seye perspective”. A motion map may for example be a 2d gridrepresentation of motion data. A grid representation of a motion map maybe based on a regular or irregular grid, i.e. the cells do notnecessarily have to be square, but can also change their size, shapeetc. In the embodiments, the cells of the grid cover the whole mapdomain. A motion map may for example store velocity data in dependenceof generic x, z coordinates.

Data association may be given by the assignment of data to cells of themotion map. Each cell may be defined by a specific x, z coordinate pair.

The circuitry may be configured to fuse the data of a Doppler radar anda stereo camera into a common representation. The apparatus may howevernot only be applied for the combination of Doppler radar and scene flow(stereo camera), but also for a ultrasound or LiDAR sensor and stereocamera, or other combinations, if they can be configured to providemotion information for the environment.

The circuitry may be configured to transfer to the common representationmotion information obtained from a Doppler radar.

Motion information obtained from the Doppler radar may comprise polarcoordinates of a cell in the polar coordinate space and angular andradial components of the velocity attributed to the cell.

The circuitry may be configured to transfer to the common representationmotion information obtained by scene flow estimation on images capturedby a stereo camera.

Motion information obtained from scene flow estimation may compriseimage positions, disparity data and displacement data.

The circuitry may be configured to reconstruct position and velocityfrom sensor data to obtain a motion map of the sensor data.

The circuitry may be configured to apply an error model for motion onthe processed data to obtain a motion map of the sensor data. An errormodel may for example determine mean and covariance of sensormeasurements.

The circuitry may for example be configured to represent the motion mapby μ_(v) _(x) _(,v) _(z) , Σ_(v) _(x) _(,v) _(z) ; x, z where x, z are2d Cartesian coordinates in the motion map, μ_(v) _(x) _(,v) _(z) arethe mean values of a normal distribution describing a velocity estimate(v_(x), v_(z)), and Σ_(v) _(x) _(,v) _(z) is the covariance matrix thatrepresents the uncertainty of the motion measurement obtained from asensor-specific error model.

The circuitry may be configured to fuse sensor data cell-wise bydetermining for each cell a joint probability for the velocity.

The circuitry may be configured to assume, for every grid cell of amotion map, that the velocity v of contacts in this cell follows anormal distribution.

The circuitry may be configured to transform parameters of an errormodel into an auxiliary representation. This auxiliary representationmay for example be used in implementations to speed up or otherwisesimplify the processing.

In general, the circuitry may be configured to fuse information fromsensors “a” and “b” given velocity distributions in the same cell.Sensors “a” and “b” may be of any one of the sensor types describedabove or in the embodiments described below in more detail. This can beextended to any finite number of sensors of any kind that provide motioninformation that can be converted to a motion map.

The embodiments also disclose a method comprising transferring motioninformation obtained from a plurality of sensors to a commonrepresentation. The method may comprise anyone of the processing stepsdescribed above or in the embodiments described below in more detail.

The embodiments also disclose a computer program comprising instructionswhich when, carried out on a processor, cause the processor to transfermotion information obtained from a plurality of sensors to a commonrepresentation. The computer program may comprise instructions toperform any one of the processing steps described above or in theembodiments described below in more detail.

The embodiments also disclose a tangible computer-readable medium thatstores a computer program comprising instructions which, when carriedout on a processor, cause the processor to transfer motion informationobtained from a plurality of sensors to a common representation. Thetangible computer-readable medium may store a computer programcomprising instructions to perform anyone of the processing stepsdescribed above or in the embodiments described below in more detail.

The embodiments are now described in more detail with reference to theaccompanying drawings.

FIG. 1 schematically describes the generation of a fused motion map byfusing data of a Doppler radar and data obtained from scene flowestimation based on images of a stereo camera. At 101, a Doppler radargenerates raw data 111. At 102, the raw data is processed to produceprocessed data 112 such as range, angle and speed data by calibration,FFT, beam-forming, etc. The processed data 112 may for example be atwo-dimensional array in polar coordinates r, φ, or a three-dimensionalarray with axes related to polar position coordinates r and φ as well asradial velocity v_(r). At 103, the processed data 112 is aligned to amotion map 113. This alignment transfers the motion informationcontained in the processed data to a common representation. At 201, astereo camera produces raw images 211. At 202, the raw images 211 areprocessed by calibration, stereo & motion estimation (scene flowestimation), etc. whereby processed data 212 in the form of 3d pointsand 3d velocities are produced. At 203, the processed data 212 arealigned to generate a motion map 204. This alignment transfers themotion information obtained from camera images to a commonrepresentation. Both motion maps 104 and 204 are fused in 301 to createa fused motion map 302 in which both data are combined. Data associationis given by the assignment of data to cells of the motion map duringdata alignment in 103 and 203.

FIG. 2 shows in more detail the alignment 103 of the processed data 112from the Doppler radar in order to generate motion map 104. Theprocessed data 112 is represented as (v_(ϕ), v_(r); ϕ, r), where ϕ, rdenote the polar coordinates of a cell in the polar coordinate space (ϕ,r) and v_(ϕ), v_(r) are the angular and radial component of the velocityattributed to the cell. As state-of-the-art radars do not provideinformation on v_(ϕ), the v_(ϕ) component can be modelled to be zero andto have an infinite uncertainty. Alternatively, for Doppler radar, v_(ϕ)could also be removed from the model. In the alignment process 103 aposition/velocity reconstruction and an error model for motion isapplied to the processed data 112 to obtain the motion map 104. Themotion map space is represented by spatial coordinates (x, z). Data inthe motion map is represented by (μ_(v) _(x) _(,v) _(z) , Σ_(v) _(x)_(,v) _(z) ; x, z) where x, z are 2d Cartesian coordinates in the motionmap, μ_(v) _(x) _(,v) _(z) are the mean values of a normal distributiondescribing a velocity estimate (v_(x), v_(z)), and Σ_(v) _(x) _(,v) _(z)is the respective covariance matrix that represents the uncertainty ofthe motion measurement obtained from a sensor-specific error model. Asrepresented by 105 in FIG. 2, with a Doppler radar, the error μ_(v) _(x)_(,v) _(z) in the radial direction of the velocity space (v_(x), v_(z))is smaller than the error Σ_(v) _(x) _(,v) _(z) in the angular directionof the velocity space (v_(x), v_(z)).

FIG. 3 shows in more detail the alignment 203 of the processed data 212from the stereo camera in order to generate motion map 204. Theprocessed data 212 is represented as (D, D′, μ_(x), μ_(y); p_(x), p_(y))where p_(x), p_(y) are image positions (pixel coordinates) in the imagespace (p_(x), p_(y)), and D represents disparity data at a first pointin time, D′ represents disparity data at a second point in time, andμ_(x), μ_(y) represents the displacement of a point in a camera view(e.g. left camera). In the alignment process 203 a 3d position/velocityreconstruction, a projection to motion map coordinates, and an errormodel for motion is applied to the processed data 212 to obtain themotion map 204. The motion map space is represented by spatialcoordinates (x, z). Again, data in the motion map is represented by(μ_(v) _(x) _(,v) _(z) , Σ_(v) _(x) _(,v) _(z) ; x, z) where x, z are 2dCartesian coordinates in the motion map, μ_(v) _(x) _(,v) _(z) are themean values of a normal distribution describing a velocity estimate(v_(x), v_(z)), and Σ_(v) _(x) _(,v) _(z) is the covariance matrix thatrepresents the uncertainty of the motion measurement obtained from asensor-specific error model. As represented by 205 in FIG. 3, with astereo camera, the uncertainty of the velocity is typically smaller inthe angular direction compared to the radial direction in the velocityspace.

FIG. 4 shows in more detail the process 301 of fusing the motion map 104obtained from data of a Doppler radar and the motion map 204 obtainedfrom data of a scene flow analysis of stereo camera images in order toobtain a fused motion map 302. According to this example, the motion map104 is a 2d grid representation in spatial coordinates (x, z) of theprocessed data 212 after alignment and the motion map 204 is a 2d gridrepresentation in spatial coordinates (x, z) of the processed data 212after alignment. Fusion 301 merges the information cell-wise bydetermining for each cell a joint probability for the velocity (v_(x),v_(z)). The fused motion map 302 is, again, a 2d grid representation inspatial coordinates (x, z).

FIG. 5a schematically shows the uncertainty of a single velocitymeasurement as seen by a scene flow analysis (stereo camera). The figureshows the spatial resolution uncertainty of the observed velocity of thestereo camera in the associated motion map. The motion map stores incell 214 the 2d mean value and the uncertainty of the z velocity vector,indicated by a probability distribution 215. The probabilitydistribution 215 represents a sensor-specific error model that encodes2d velocity uncertainty as a covariance matrix Σ_(v) _(x) _(,v) _(z) .The actual velocity of an object in cell 214 is indicated by velocityvector 211. A velocity measurement of the stereo camera is indicated asa measured radial velocity component 212 and a measured angular velocitycomponent 213. The stereo camera-based scene flow analysis has a bettervelocity precision in angular direction than in radial direction so thatthe uncertainty in the radial direction is larger than in the angulardirection. Accordingly, the probability distribution 215 is representedin FIG. 5a by an ellipse that is narrower in radial direction than inangular direction.

FIG. 5b schematically shows the uncertainty of a single velocitymeasurement as seen by a Doppler radar. The figure shows the spatialresolution uncertainty of the Doppler radar in the associated motionmap. The motion map stores in cell 114 an x-z velocity vector and itsrespective uncertainty, indicated by a probability distribution 115. Theprobability distribution 115 represents a sensor-specific error modelthat encodes 2d velocity uncertainty as a covariance matrix Σ_(v) _(x)_(,v) _(z) . The actual velocity of an object in cell 114 is indicatedby velocity vector 111. A velocity measurement of the Doppler radar isindicated as a measured radial velocity component 112. The Doppler radarhas a good velocity resolution in radial direction but the angularvelocity component is typically not observable. Accordingly, theprobability distribution 215 is represented in FIG. 5b by an ellipsethat is narrow in radial direction but very large in angular direction.

FIG. 5c shows the reduced velocity measurement uncertainty in a fusedmotion map generated by fusing the aligned data of a Doppler radar and astereo camera. The figure shows the uncertainty of the spatialresolution of the fused data in the fused motion map. The fused motionmap stores in cell 314 an x-z velocity vector and its respectiveuncertainty, indicated by a probability distribution 315. The actualvelocity of an object in cell 314 is indicated by velocity vector 311. Afused velocity measurement of the stereo camera is indicated as a radialvelocity component 312 and an angular velocity component 313. As thecamera and radar based measurements provide information on motion withcomplementary precision, the fused data has a good velocity resolutionin radial direction and in axial direction. Accordingly, the probabilitydistribution 315 of the fused measurement is represented in FIG. 5c byan ellipse that is narrow in radial direction and narrow in angulardirection. The result is (under ideal conditions) a measurement with lowuncertainty in both radial and angular direction.

FIG. 6 shows an example of a motion map derived from Doppler radarmeasurements. In this visualization the brightness indicates velocityvalue. In alternative visualizations of a motion map, the informationstored for each cell might be color-encoded as following: colorsaturation indicates the precision (unsaturated=zero precision), andbrightness and hue indicate velocity value (black=zero speed) anddirection (blue=up, yellow=down, magenta=left, green=right),respectively. The Doppler radar has a good velocity resolution in radialdirection but the angular velocity component is not observable.

FIG. 7 shows an example of a motion map derived from scene flowmeasurements based on images captured by a stereo camera. The scene flowanalysis has a better velocity resolution in angular direction than inradial direction, so that the uncertainty in the radial direction islarger than in the angular direction.

FIG. 8 shows an example of a fused motion map that combines the motionmap of FIG. 6 obtained by a Doppler radar with the motion map of FIG. 7obtained by scene flow analysis based on images captured by a stereocamera. The result is a measurement with low uncertainty in both radialand angular direction.

Motion Map Representation

For every grid cell we assume that the velocities v of contacts in thiscell follow a normal distribution:

$v = {\begin{pmatrix}v_{x} \\v_{z}\end{pmatrix} \sim {N\left( {\mu,\sum} \right)}}$That is, for each cell of the motion map we have one speed estimate fromscene flow(s) and Doppler radar (r) each, encoded as 2d normaldistribution:p(v|s)=N(μ_(s),Σ_(s)) and p(v/r)=N(μ_(r),Σ_(r))It should be noted here that a scene flow speed estimate as describedabove may possibly be fused from several scene flow measurements, andthat a Doppler radar estimate as described above may possibly be fusedfrom several Doppler radar measurements.

“Auxiliary” Representation of Motion Map

According to an alternative representation which is, in implementation,more efficient for fusion (in the following called “auxiliary”representation) the parameters μ, Σ of the normal distribution arerepresented as follows:P=Σ ⁻¹ andb=Σ ⁻¹ μ=PμAccordingly, the covariance matrix is represented by a precisionP:=Σ⁻¹∈R^(2×2) which is positively semidefinite and by b:=Σ⁻¹μ∈R².

The conversion from the “auxiliary” representation to the normalrepresentation is performed by:Σ=P ⁻¹ ,μ=Σb

Motion Map Fusion

Assuming two motion maps comprising information from sensors “a” and“b”, given two distributions for v in the same cell:p(v|a)=N(μ_(a),Σ_(a)) and p(v|b)=N(μ_(b),Σ_(b))where μ_(a), μ_(b) are the respective mean values of the normaldistribution N and Σ_(a), Σ_(b), are the respective covariance matricesof the normal distribution N.

These distributions are fused asp(v|a,b)=N(μ_(c),Σ_(c))∝p(v|a)p(v|b)=N(μ_(a),Σ_(a))N(μ_(b),Σ_(b))withΣ_(c)=(Σ_(a) ⁻¹+Σ_(b) ⁻¹)⁻¹μ_(c)Σ_(c)(Σ_(a) ⁻¹μ_(a)+Σ_(b) ⁻¹μ_(b))In the auxiliary representation, these distributions are represented by:p _(a) ,b _(a), and P _(b) ,P _(b)and using the auxiliary representation, the fusing reads as:P _(c) =P _(a) +P _(b) and b _(c) =b _(a) +b _(b)

Transformation of Doppler Radar Information to Motion Map

In the following, the process of aligning (103 in FIG. 1) the processeddata (112 in FIG. 1) of the Doppler radar to obtain a motion map (104 inFIG. 1) of the Doppler radar data is explained in more detail.

For a given radar position (r, ϕ) in polar coordinates all consideredradial velocities are provided with a weight q(v_(r)) derived from thedetection power. This information is encoded in a range-angle-speed cubeand during the aligning process, for each range-angle-pair, the radialvelocity v_(r) is computed as q-weighted average along thespeed-dimensions, and there is computed a precision estimate σ_(Vr) ⁻²of this information as RMS of q along the speed information.

Then, for each x-z-cell of the motion map the radial velocity andprecision from the range-angle-map are interpolated, and the speedinformation is encoded as a 2d normal distribution with mean μ_(v):speed v_(r) in radial direction, speed 0 in angular direction andcovariance Σ_(v): eigenvalues: σ_(Vr) ² in radial direction, ≈∞ inangular direction.

For example, a motion model may be defined as follows:e: =(sin(ϕ), cos(ϕ))μ_(P) :=e*rμ_(V) :=e*v _(r)Σ_(P) ⁻¹:=(e*e ^(τ))*σ_(Pr) ⁻²+(I−e*e ^(τ))*σ_(Pϕ) ⁻²Σ_(V) ⁻¹ :=e*e ^(τ)*σ_(Vr) ⁻²

Here, σ_(Pϕ) ⁻² and σ_(Pr) ⁻² are parameters that define the precisionof the position measurement and σ_(Vr) ⁻² is a parameter that definesthe precision of the velocity measurements and is derived from q asdescribed above. These precisions may be defined by the model accordingto the capabilities of the sensor. Here, we assume for simplicity:σ_(Pϕ) ⁻²:=1σ_(Pr) ⁻²:=1

However, in practice, any other definition may be used that representsthe capabilities of the sensor in a better way.

After that, a position (x, z) in the coordinate system of the motion mapis determined and the mean value and covariance for the normaldistribution that define the speed are calculated as follows:μ_(p) :=Aμ _(P) +bμ_(v) :=Aμ _(V) +bΣ_(p) :=AΣ _(P) A ^(τ)Σ_(v) :=AΣ _(V) A ^(τ)

Here, A∈

^(2×2) is a predefined correction matrix and b∈

² is a predefined correction vector that correct for the relative sensoralignment.

Next, μ_(v) and Σ_(v) are converted to the auxiliary representation:P _(v)=Σ_(v) ⁻¹ and b _(v) =P _(v)·μ_(v)

Finally, for each cell i with center position (x_(i), z_(i)) in themotion map coordinate system, there is computed a weightw _(i) =N(x _(i) ,z _(i);μ_(p),Σ_(vp))

To iteratively accumulate information from multiple measurements foreach cell a joint probability is computed by updating the velocityestimate (P_(i), b_(i)) in cell i with the new information (w_(i)P_(v),w_(i)b_(v)) which is the information derived from the currentlyconsidered measurement associated to cell i. This amounts to computingthe joint probability asP _(i,new) =P _(i) +w _(i) P _(v)b _(i,new) =b _(i) +w _(i) b _(v)

The information in cell i is then replaced by (P_(i,new), b_(i,new)).

Here, b_(i) and P_(i) may be initialized as (0,0) and

$\begin{pmatrix}0 & 0 \\0 & 0\end{pmatrix}.$

Transformation of Scene Flow Information to Motion Map

In the following, the process of aligning (203 in FIG. 1) the processeddata (212 in FIG. 1) of the scene flow (stereo camera) measurements toobtain a motion map (204 in FIG. 1) of the scene flow data is explainedin more detail.

Given an image position (p_(x), p_(y)) in the image domain of thecamera, measurements in this position comprise:

-   -   3d position μ_(P):=(μ_(P) _(x) , μ_(P) _(y) , μ_(P) _(z) )    -   3d velocity μ_(V):=(μ_(V) _(x) , μ_(V) _(y) , μ_(V) _(z) )

All this information is given in a camera coordinate system, e.g. theleft camera.

Based on this information, irrelevant contacts (road surface, sufficientheadroom) are removed from the data. Only measurements for example 0 to3 m above road level are kept and a simple road model (pre-defined 3dplane) is used. To this end, the height of a point above the street isdetermined. If the point is on or below the street surface, then thismeasurement is ignored. If the point is above a safety headroom height,then the measurement is ignored. Otherwise, the measurement ismaintained.

Then, 3d position and velocity measurements are projected to the 2dmotion map using a simple road model. Also, alignment between camera andradar is corrected. A position (x, z) in the coordinate system of themotion map is determined, e.g. by assuming a street plane and projectingthe model to 2×3 matrix A, 2×1 vector b asμ_(x) :=A μ _(P) +bμ_(v) :=A μ _(V) +b

Here, A, b also correct for the relative sensor alignment.

The measurements in combination with a 3d error model for position andspeed (see section “Error Model for Scene Flow” below) also provideΣ_(P) and Σ_(V), both 3×3 covariance matrices, which representsensor-specific precision. For example, each measurement is encodedusing a normal distribution. The mean μ is given by the measurement. Thecovariance Σ is given by the error model. Σ_(P) and Σ_(V), both 3×3covariance matrices, represent the sensor-specific precision:Σ_(x) :=AΣ _(P) A ^(τ)Σ_(v) :=AΣ _(V) A ^(τ)

Then μ_(v) and Σ_(v) are converted to the auxiliary representation:P _(v)=Σ_(v) ⁻¹ and b _(v) =P _(v)μ_(v)

Finally, for each cell i with center position (x_(i), z_(i)) in themotion map coordinate system, it is computed a weightw _(i) =N(x _(i) ,z _(i),μ_(x),Σ_(x))

To iteratively accumulate information from multiple measurements foreach cell a joint probability is computed by updating the velocityestimate (P_(i), b_(i)) in cell i with the new information (w_(i)P_(v),w_(i)b_(v)) which is the information derived from the currentlyconsidered measurement associated to cell i. This amounts to computingthe joint probability asP _(i,new) =P _(i) +w _(i) P _(v)b _(i,new) =b _(i) +w _(i) b _(v)

The information in cell i is then replaced by (P_(i,new), b_(i,new)).Here, b_(i) and P_(i) are initialized as (0,0) and

$\begin{pmatrix}0 & 0 \\0 & 0\end{pmatrix}.$

Error Model for Scene Flow

For the 3d error model used in the transformation of scene flowinformation to a motion map as described above, the computation ofnormal distributions for 3d position (μ_(p), Σ_(p)) and 3d velocity(μ_(V), Σ_(V)) assumptions is as follows:

Information is recorded from four camera views: An image is obtainedfrom a first (e.g. left) camera view C_(L) and an image is obtained froma second (e.g. right) camera view C_(R) at their position at a time t.Still further, an image is obtained from a first camera view C′_(L) andfrom a second camera view C′_(R) at their position at a time t+T, whereT is the time difference between the camera views.

Then, the data from camera pairs (C_(L), C_(R)) and (C′_(L), C′_(R)) iscalibrated and rectified such that the baselines b (between C_(L) andC_(R)) and b′ (between C′_(L) and C′_(R)) are known, and the intrinsiccalibration K (valid for C_(L) and C_(R)) and K′ (valid for C′_(L) andC′_(R)) are known.

Given an image position (p_(x), p_(y)) in the image domain of the cameraand measurements in this position, a 3d position μ_(P):=(μ_(P) _(x) ,μ_(P) _(y) , μ_(P) _(z) ) and a 3d velocity μ_(V):=(μ_(V) _(x) , μ_(V)_(y) , μ_(V) _(z) ) are obtained for every image position.

From the measurements, per pixel at position (p_(x), p_(y)) in cameraC_(L) the following information is given:

-   -   disparity D between a point (p_(x), p_(y)) in C_(L) and        corresponding position (p_(x)+D, p_(y)) in C_(R)    -   displacement (u_(x), u_(y)) between a point (p_(x), p_(y)) in        C_(L) and the corresponding position (p_(x)+u_(x), p_(y)+u_(y))        in C′_(L)    -   disparity D′ between a point (p_(x)+u_(x), p_(y)+u_(y)) in        C′_(L) and the corresponding point (p_(x)+u_(x)+D′, p_(y)+u_(y))        in C′_(R)

Still further, covariance estimates Σ_(m) and Σ′_(m) for measurementerrors for (D, p_(x), p_(y)) and (D′, p_(x)+u_(x), p_(y)+u_(y)) aredetermined, e.g. from experimental measurements.

Based on this information the following computations are performed:

There is computed (using ƒ as defined below) and returned a distributionfor 3d position P, given by(μ_(P),Σ_(P)):=ƒ(D,p _(x) ,p _(y) ,K,b,Σ _(m)),where(μ,Σ)=ƒ(D,p _(x) ,p _(y) ,K,b,Σ _(D,p) _(x) _(,p) _(y) )

Still further, there is computed a distribution for P′, given by(μ_(P′),Σ_(P′)):=ƒ(D′,p _(x) u _(x) ,p _(y) u _(y) ,K′,b′,Σ′ ^(m)).

And there is computed a distribution of the 3d velocity V=T⁻¹ (P′−P)˜N(μ_(V), Σ_(V)), given by (linear combination of random variables):μ_(V) :=T ⁻¹(μ_(P′)−μ_(P))Σ_(V) =T ⁻²(Σ_(P′)+Σ_(P))

-   -   The previous computations utilize an auxiliary function ƒ (D,        p_(x), p_(y), K, b, Σ_(D,p) _(x) _(,p) _(y) ) which is computed        as follows. Given: the pixel position (p_(x), p_(y)) in the        first camera,    -   the disparity D between a point (p_(x), p_(y)) in the first        camera and corresponding position (p_(x)+D, p_(y)) in a second        camera,    -   3×3 matrix K, describing the intrinsic camera parameters of both        the first and second camera,    -   baseline b describing the distance between the origins of the        coordinate systems of the first and second camera, and    -   3×3 covariance matrix Σ_(D,p) _(x) _(,p) _(y) describing an        estimate of the errors in D, p_(x) and P_(y),        there is computed the parameters of an approximate normal        distribution of a point {circumflex over (X)} reconstructed via        triangulation between the first and second camera.

The point {circumflex over (X)} is reconstructed in the coordinatesystem of the first camera, such that {circumflex over (X)}˜N(μ, Σ),with parameters computed as follows:

-   -   We introduce an auxiliary function:

${X\left( {d,x,y} \right)} \approx {\left( {b\mspace{20mu} K_{1,1}} \right)d^{- 1}{K^{- 1}\begin{pmatrix}x \\y \\1\end{pmatrix}}}$(stereo reconstruction via triangulation)

-   -   Then assuming (D, p_(x), p_(y)) to be normally distributed with        Σ_(D,p) _(x) _(,p) _(y) we get (error propagation):        μ:=X(D,p _(x) ,p _(y)) and Σ:=J _(X)(D,p _(x) ,p        _(y))Σ_(D,x,y)(J _(X)(D,p _(x) ,p _(y)))^(τ)

where J_(X)(D, p_(x), p_(y)) is the 3×3 Jacobian matrix of X(d, x, y)evaluated at position (D, p_(x), p_(y)).

Here, the Jacobian matrix J_(X)(d, x, y) is given by (first ordercomponent-wise derivatives)

${J_{X}\left( {d,x,y} \right)} = {\left( {b\mspace{14mu} K_{1,1}} \right){K^{- 1}\begin{bmatrix}{\begin{pmatrix}x \\y \\1\end{pmatrix}\left( {- d^{- 2}} \right)} & {\begin{pmatrix}1 \\0 \\0\end{pmatrix}d^{- 1}} & {\begin{pmatrix}0 \\1 \\0\end{pmatrix}d^{- 1}}\end{bmatrix}}}$

The thus obtained parameters μ, Σ returned by f can now be used in thetransformation of scene flow information to a motion map as describedabove.

Implementation

The technology according to the embodiments of the present disclosure isapplicable to various products. For example, the technology according toan embodiment of the present disclosure may be implemented as a deviceincluded in a mobile body that is any of kinds of automobiles, electricvehicles, hybrid electric vehicles, motorcycles, bicycles, personalmobility vehicles, airplanes, drones, ships, robots, constructionmachinery, agricultural machinery (tractors), and the like.

FIG. 9 is a block diagram depicting an example of schematicconfiguration of a vehicle control system 7000 as an example of a mobilebody control system to which the technology according to an embodimentof the present disclosure can be applied. The vehicle control system7000 includes a plurality of electronic control units connected to eachother via a communication network 7010. In the example depicted in FIG.9, the vehicle control system 7000 includes a driving system controlunit 7100, a body system control unit 7200, a battery control unit 7300,an outside-vehicle information detecting unit 7400, an in-vehicleinformation detecting unit 7500, and an integrated control unit 7600.The communication network 7010 connecting the plurality of control unitsto each other may, for example, be a vehicle-mounted communicationnetwork compliant with an arbitrary standard such as controller areanetwork (CAN), local interconnect network (LIN), local area network(LAN), FlexRay (registered trademark), or the like.

Each of the control units includes: a microcomputer that performsarithmetic processing according to various kinds of programs; a storagesection that stores the programs executed by the microcomputer,parameters used for various kinds of operations, or the like; and adriving circuit that drives various kinds of control target devices.Each of the control units further includes: a network interface (I/F)for performing communication with other control units via thecommunication network 7010; and a communication I/F for performingcommunication with a device, a sensor, or the like within and withoutthe vehicle by wire communication or radio communication. A functionalconfiguration of the integrated control unit 7600 illustrated in FIG. 9includes a microcomputer 7610, a general-purpose communication I/F 7620,a dedicated communication I/F 7630, a positioning section 7640, a beaconreceiving section 7650, an in-vehicle device I/F 7660, a sound/imageoutput section 7670, a vehicle-mounted network I/F 7680, and a storagesection 7690. The other control units similarly include a microcomputer,a communication I/F, a storage section, and the like.

The driving system control unit 7100 controls the operation of devicesrelated to the driving system of the vehicle in accordance with variouskinds of programs. For example, the driving system control unit 7100functions as a control device for a driving force generating device forgenerating the driving force of the vehicle, such as an internalcombustion engine, a driving motor, or the like, a driving forcetransmitting mechanism for transmitting the driving force to wheels, asteering mechanism for adjusting the steering angle of the vehicle, abraking device for generating the braking force of the vehicle, and thelike. The driving system control unit 7100 may have a function as acontrol device of an antilock brake system (ABS), electronic stabilitycontrol (ESC), or the like.

The driving system control unit 7100 is connected with a vehicle statedetecting section 7110. The vehicle state detecting section 7110, forexample, includes at least one of a gyro sensor that detects the angularvelocity of axial rotational movement of a vehicle body, an accelerationsensor that detects the acceleration of the vehicle, and sensors fordetecting an amount of operation of an accelerator pedal, an amount ofoperation of a brake pedal, the steering angle of a steering wheel, anengine speed or the rotational speed of wheels, and the like. Thedriving system control unit 7100 performs arithmetic processing using asignal input from the vehicle state detecting section 7110, and controlsthe internal combustion engine, the driving motor, an electric powersteering device, the brake device, and the like.

The body system control unit 7200 controls the operation of variouskinds of devices provided to the vehicle body in accordance with variouskinds of programs. For example, the body system control unit 7200functions as a control device for a keyless entry system, a smart keysystem, a power window device, or various kinds of lamps such as aheadlamp, a backup lamp, a brake lamp, a turn signal, a fog lamp, or thelike. In this case, radio waves transmitted from a mobile device as analternative to a key or signals of various kinds of switches can beinput to the body system control unit 7200. The body system control unit7200 receives these input radio waves or signals, and controls a doorlock device, the power window device, the lamps, or the like of thevehicle.

The battery control unit 7300 controls a secondary battery 7310, whichis a power supply source for the driving motor, in accordance withvarious kinds of programs. For example, the battery control unit 7300 issupplied with information about a battery temperature, a battery outputvoltage, an amount of charge remaining in the battery, or the like froma battery device including the secondary battery 7310. The batterycontrol unit 7300 performs arithmetic processing using these signals,and performs control for regulating the temperature of the secondarybattery 7310 or controls a cooling device provided to the battery deviceor the like.

The outside-vehicle information detecting unit 7400 detects informationabout the outside of the vehicle including the vehicle control system7000. For example, the outside-vehicle information detecting unit 7400is connected with at least one of an imaging section 7410 and anoutside-vehicle information detecting section 7420. The imaging section7410 includes at least one of a time-of-flight (ToF) camera, a stereocamera, a monocular camera, an infrared camera, and other cameras. Theoutside-vehicle information detecting section 7420, for example,includes at least one of an environmental sensor for detecting currentatmospheric conditions or weather conditions and a peripheralinformation detecting sensor for detecting another vehicle, an obstacle,a pedestrian, or the like on the periphery of the vehicle including thevehicle control system 7000.

The environmental sensor, for example, may be at least one of a raindrop sensor detecting rain, a fog sensor detecting fog, a sunshinesensor detecting a degree of sunshine, and a snow sensor detectingsnowfall. The peripheral information detecting sensor may be at leastone of an ultrasonic sensor, a radar device, and a LiDAR device (LightDetection and Ranging device, or Laser imaging detection and rangingdevice). Each of the imaging section 7410 and the outside-vehicleinformation detecting section 7420 may be provided as an independentsensor or device, or may be provided as a device in which a plurality ofsensors or devices are integrated.

FIG. 10 depicts an example of installation positions of the imagingsection 7410 and the outside-vehicle information detecting section 7420.Imaging sections 7910, 7912, 7914, 7916, and 7918 are, for example,disposed at at least one of positions on a front nose, sideview mirrors,a rear bumper, and a back door of the vehicle 7900 and a position on anupper portion of a windshield within the interior of the vehicle. Theimaging section 7910 provided at the front nose and the imaging section7918 provided at the upper portion of the windshield within the interiorof the vehicle obtain mainly an image of the front of the vehicle 7900.The imaging sections 7912 and 7914 provided at the sideview mirrorsobtain mainly an image of the sides of the vehicle 7900. The imagingsection 7916 provided at the rear bumper or the back door obtains mainlyan image of the rear of the vehicle 7900. The imaging section 7918provided at the upper portion of the windshield within the interior ofthe vehicle is used mainly to detect a preceding vehicle, a pedestrian,an obstacle, a signal, a traffic sign, a lane, or the like.

Incidentally, FIG. 10 depicts an example of photographing ranges of therespective imaging sections 7910, 7912, 7914, and 7916. An imaging rangea represents the imaging range of the imaging section 7910 provided atthe front nose. Imaging ranges b and c respectively represent theimaging ranges of the imaging sections 7912 and 7914 provided at thesideview mirrors. An imaging range d represents the imaging range of theimaging section 7916 provided at the rear bumper or the back door. Abird's-eye image of the vehicle 7900 as viewed from above can beobtained by superimposing image data imaged by the imaging sections7910, 7912, 7914, and 7916, for example.

Outside-vehicle information detecting sections 7920, 7922, 7924, 7926,7928, and 7930 provided at the front, rear, sides, and corners of thevehicle 7900 and the upper portion of the windshield within the interiorof the vehicle may be, for example, an ultrasonic sensor or a radardevice. The outside-vehicle vehicle information detecting sections 7920,7926, and 7930 provided at the front nose of the vehicle 7900, the rearbumper, the back door of the vehicle 7900, and the upper portion of thewindshield within the interior of the vehicle may be a LiDAR device, forexample. These outside-vehicle information detecting sections 7920 to7930 are used mainly to detect a preceding vehicle, a pedestrian, anobstacle, or the like.

Returning to FIG. 9, the description will be continued. Theoutside-vehicle information detecting unit 7400 causes the imagingsection 7410 to produce an image of the outside of the vehicle, andreceives produced image data. In addition, the outside-vehicleinformation detecting unit 7400 receives detection information from theoutside-vehicle information detecting section 7420 connected to theoutside-vehicle information detecting unit 7400. In a case where theoutside-vehicle information detecting section 7420 is an ultrasonicsensor, a radar device, or a LIDAR device, the outside-vehicleinformation detecting unit 7400 transmits an ultrasonic wave, anelectromagnetic wave, or the like, and receives information of areceived reflected wave. On the basis of the received information, theoutside-vehicle information detecting unit 7400 may perform processingof detecting an object such as a human, a vehicle, an obstacle, a sign,a character on a road surface, or the like, or processing of detecting adistance thereto. The outside-vehicle information detecting unit 7400may perform environment recognition processing of recognizing rain, fog,road surface conditions, or the like on the basis of the receivedinformation. The outside-vehicle information detecting unit 7400 maycalculate a distance to an object outside the vehicle on the basis ofthe received information.

In addition, on the basis of the received image data, theoutside-vehicle information detecting unit 7400 may perform imagerecognition processing of recognizing a human, a vehicle, an obstacle, asign, a character on a road surface, or the like, or processing ofdetecting a distance thereto. The outside-vehicle information detectingunit 7400 may subject the received image data to processing such asdistortion correction, alignment, or the like, and combine the imagedata imaged by a plurality of different imaging sections 7410 togenerate a bird's-eye image or a panoramic image. The outside-vehicleinformation detecting unit 7400 may perform viewpoint conversionprocessing using the image data imaged by the imaging section 7410including the different imaging parts.

The in-vehicle information detecting unit 7500 detects information aboutthe inside of the vehicle. The in-vehicle information detecting unit7500 is, for example, connected with a driver state detecting section7510 that detects the state of a driver. The driver state detectingsection 7510 may include a camera that images the driver, a biosensorthat detects biological information on the driver, a microphone thatcollects sound within the interior of the vehicle, or the like. Thebiosensor is, for example, disposed in a seat surface, the steeringwheel, or the like, and detects biological information on an occupantsitting in a seat or the driver holding the steering wheel. On the basisof detection information input from the driver state detecting section7510, the in-vehicle information detecting unit 7500 may calculate adegree of fatigue of the driver or a degree of concentration of thedriver, or may determine whether the driver is dozing. The in-vehicleinformation detecting unit 7500 may subject an audio signal obtained bythe collection of the sound to processing such as noise cancelingprocessing or the like.

The integrated control unit 7600 controls general operation within thevehicle control system 7000 in accordance with various kinds ofprograms. The integrated control unit 7600 is connected with an inputsection 7800. The input section 7800 is implemented by a device capableof input operation by an occupant, such, for example, as a touch panel,a button, a microphone, a switch, a lever, or the like. The integratedcontrol unit 7600 may be supplied with data obtained by voicerecognition of voice input through the microphone. The input section7800 may, for example, be a remote control device using infrared rays orother radio waves, or an external connecting device such as a mobiletelephone, a personal digital assistant (PDA), or the like that supportsoperation of the vehicle control system 7000. The input section 7800 maybe, for example, a camera. In that case, an occupant can inputinformation by gesture. Alternatively, data may be input which isobtained by detecting the movement of a wearable device that an occupantwears. Further, the input section 7800 may, for example, include aninput control circuit or the like that generates an input signal on thebasis of information input by an occupant or the like using theabove-described input section 7800, and which outputs the generatedinput signal to the integrated control unit 7600. An occupant or thelike inputs various kinds of data or gives an instruction for processingoperation to the vehicle control system 7000 by operating the inputsection 7800.

The storage section 7690 may include a read only memory (ROM) thatstores various kinds of programs executed by the microcomputer and arandom access memory (RAM) that stores various kinds of parameters,operation results, sensor values, or the like. In addition, the storagesection 7690 may be implemented by a magnetic storage device such as ahard disc drive (HDD) or the like, a semiconductor storage device, anoptical storage device, a magneto-optical storage device, or the like.

The general-purpose communication I/F 7620 is a communication I/F usedwidely, which communication I/F mediates communication with variousapparatuses present in an external environment 7750. The general-purposecommunication I/F 7620 may implement a cellular communication protocolsuch as global system for mobile communications (GSM (registeredtrademark)), worldwide interoperability for microwave access (WiMAX(registered trademark)), long term evolution (LTE (registeredtrademark)), LTE-advanced (LTE-A), or the like, or another wirelesscommunication protocol such as wireless LAN (referred to also aswireless fidelity (Wi-Fi (registered trademark)), Bluetooth (registeredtrademark), or the like. The general-purpose communication I/F 7620 may,for example, connect to an apparatus (for example, an application serveror a control server) present on an external network (for example, theInternet, a cloud network, or a company-specific network) via a basestation or an access point. In addition, the general-purposecommunication I/F 7620 may connect to a terminal present in the vicinityof the vehicle (which terminal is, for example, a terminal of thedriver, a pedestrian, or a store, or a machine type communication (MTC)terminal) using a peer to peer (P2P) technology, for example.

The dedicated communication I/F 7630 is a communication I/F thatsupports a communication protocol developed for use in vehicles. Thededicated communication I/F 7630 may implement a standard protocol suchas, for example, wireless access in vehicle environment (WAVE), which isa combination of institute of electrical and electronic engineers (IEEE)802.11p as a lower layer and IEEE 1609 as a higher layer, dedicatedshort range communications (DSRC), or a cellular communication protocol.The dedicated communication I/F 7630 typically carries out V2Xcommunication as a concept including one or more of communicationbetween a vehicle and a vehicle (Vehicle to Vehicle), communicationbetween a road and a vehicle (Vehicle to Infrastructure), communicationbetween a vehicle and a home (Vehicle to Home), and communicationbetween a pedestrian and a vehicle (Vehicle to Pedestrian).

The positioning section 7640, for example, performs positioning byreceiving a global navigation satellite system (GNSS) signal from a GNSSsatellite (for example, a GPS signal from a global positioning system(GPS) satellite), and generates positional information including thelatitude, longitude, and altitude of the vehicle. Incidentally, thepositioning section 7640 may identify a current position by exchangingsignals with a wireless access point, or may obtain the positionalinformation from a terminal such as a mobile telephone, a personalHandy-phone system (PHS), or a smart phone that has a positioningfunction.

The beacon receiving section 7650, for example, receives a radio wave oran electromagnetic wave transmitted from a radio station installed on aroad or the like, and thereby obtains information about the currentposition, congestion, a closed road, a necessary time, or the like.Incidentally, the function of the beacon receiving section 7650 may beincluded in the dedicated communication I/F 7630 described above.

The in-vehicle device I/F 7660 is a communication interface thatmediates connection between the microcomputer 7610 and variousin-vehicle devices 7760 present within the vehicle. The in-vehicledevice I/F 7660 may establish wireless connection using a wirelesscommunication protocol such as wireless LAN, Bluetooth (registeredtrademark), near field communication (NFC), or wireless universal serialbus (WUSB). In addition, the in-vehicle device I/F 7660 may establishwired connection by universal serial bus (USB), high-definitionmultimedia interface (HDMI (registered trademark)), mobilehigh-definition link (MHL), or the like via a connection terminal (and acable if necessary) not depicted in the figures. The in-vehicle devices7760 may, for example, include at least one of a mobile device and awearable device possessed by an occupant and an information devicecarried into or attached to the vehicle. The in-vehicle devices 7760 mayalso include a navigation device that searches for a path to anarbitrary destination. The in-vehicle device I/F 7660 exchanges controlsignals or data signals with these in-vehicle devices 7760.

The vehicle-mounted network I/F 7680 is an interface that mediatescommunication between the microcomputer 7610 and the communicationnetwork 7010. The vehicle-mounted network I/F 7680 transmits andreceives signals or the like in conformity with a predetermined protocolsupported by the communication network 7010.

The microcomputer 7610 of the integrated control unit 7600 controls thevehicle control system 7000 in accordance with various kinds of programson the basis of information obtained via at least one of thegeneral-purpose communication I/F 7620, the dedicated communication I/F7630, the positioning section 7640, the beacon receiving section 7650,the in-vehicle device I/F 7660, and the vehicle-mounted network I/F7680. For example, the microcomputer 7610 may calculate a control targetvalue for the driving force generating device, the steering mechanism,or the braking device on the basis of the obtained information about theinside and outside of the vehicle, and output a control command to thedriving system control unit 7100. For example, the microcomputer 7610may perform cooperative control intended to implement functions of anadvanced driver assistance system (ADAS) which functions includecollision avoidance or shock mitigation for the vehicle, followingdriving based on a following distance, vehicle speed maintainingdriving, a warning of collision of the vehicle, a warning of deviationof the vehicle from a lane, or the like. In addition, the microcomputer7610 may perform cooperative control intended for automatic driving,which causes the vehicle to travel autonomously without depending on theoperation of the driver, or the like, by controlling the driving forcegenerating device, the steering mechanism, the braking device, or thelike on the basis of the obtained information about the surroundings ofthe vehicle.

The microcomputer 7610 may generate three-dimensional distanceinformation between the vehicle and an object such as a surroundingstructure, a person, or the like, and generate local map informationincluding information about the surroundings of the current position ofthe vehicle, on the basis of information obtained via at least one ofthe general-purpose communication I/F 7620, the dedicated communicationI/F 7630, the positioning section 7640, the beacon receiving section7650, the in-vehicle device I/F 7660, and the vehicle-mounted networkI/F 7680. In addition, the microcomputer 7610 may predict danger such ascollision of the vehicle, approaching of a pedestrian or the like, anentry to a closed road, or the like on the basis of the obtainedinformation, and generate a warning signal. The warning signal may, forexample, be a signal for producing a warning sound or lighting a warninglamp.

The sound/image output section 7670 transmits an output signal of atleast one of a sound and an image to an output device capable ofvisually or acoustically notifying an occupant of the vehicle or theoutside of the vehicle. In the example of FIG. 9, an audio speaker 7710,a display section 7720, and an instrument panel 7730 are illustrated asthe output device. The display section 7720 may, for example, include atleast one of an on-board display and a head-up display. The displaysection 7720 may have an augmented reality (AR) display function. Theoutput device may be other than these devices, and may be another devicesuch as headphones, a wearable device such as an eyeglass type displayworn by an occupant or the like, a projector, a lamp, or the like. In acase where the output device is a display device, the display devicevisually displays results obtained by various kinds of processingperformed by the microcomputer 7610 or information received from anothercontrol unit in various forms such as text, an image, a table, a graph,or the like. In addition, in a case where the output device is an audiooutput device, the audio output device converts an audio signalconstituted of reproduced audio data or sound data or the like into ananalog signal, and acoustically outputs the analog signal.

Incidentally, at least two control units connected to each other via thecommunication network 7010 in the example depicted in FIG. 9 may beintegrated into one control unit. Alternatively, each individual controlunit may include a plurality of control units. Further, the vehiclecontrol system 7000 may include another control unit not depicted in thefigures. In addition, part of or all the functions performed by one ofthe control units in the above description may be assigned to anothercontrol unit. That is, predetermined arithmetic processing may beperformed by any of the control units as long as information istransmitted and received via the communication network 7010. Similarly,a sensor or a device connected to one of the control units may beconnected to another control unit, and a plurality of control units maymutually transmit and receive detection information via thecommunication network 7010.

Incidentally, a computer program for realizing the functions describedin this disclosure can be implemented in integrated control unit 7600,or the like. In addition, a computer readable recording medium storingsuch a computer program can also be provided. The recording medium is,for example, a magnetic disk, an optical disk, a magneto-optical disk, aflash memory, or the like. In addition, the above-described computerprogram may be distributed via a network, for example, without therecording medium being used.

In addition, at least part of the constituent elements of the integratedcontrol unit 7600 described with reference to FIG. 9 may be implementedin a module (for example, an integrated circuit module formed with asingle die) for the integrated control unit 7600 depicted in FIG. 9.Alternatively, the integrated control unit 7600 described with referenceto FIG. 9 may be implemented by a plurality of control units of avehicle control system as depicted in FIG. 9.

* * *

The methods as described herein are also implemented in some embodimentsas a computer program causing a computer and/or a processor and/or acircuitry to perform the method, when being carried out on the computerand/or processor and/or circuitry. In some embodiments, also anon-transitory computer-readable recording medium is provided thatstores therein a computer program product, which, when executed by aprocessor/circuitry, such as the processor/circuitry described above,causes the methods described herein to be performed.

It should be noted that the embodiments describe methods with anexemplary order of method steps. The specific order of method steps is,however, given for illustrative purposes only and should not beconstrued as binding.

It should also be noted that the division of the control or circuitry ofFIG. 9 into units 931 to 940 is only made for illustration purposes andthat the present disclosure is not limited to any specific division offunctions in specific units. For instance, at least parts of thecircuitry could be implemented by a respective programmed processor,field programmable gate array (FPGA), dedicated circuits, and the like.

All units and entities described in this specification and claimed inthe appended claims can, if not stated otherwise, be implemented asintegrated circuit logic, for example on a chip, and functionalityprovided by such units and entities can, if not stated otherwise, beimplemented by software.

In so far as the embodiments of the disclosure described above areimplemented, at least in part, using a software-controlled dataprocessing apparatus, it will be appreciated that a computer programproviding such software control and a transmission, storage or othermedium by which such a computer program is provided are envisaged asaspects of the present disclosure.

Note that the present technology can also be configured as describedbelow.

(1) An apparatus comprising circuitry configured to transfer motioninformation obtained from a plurality of sensors of different or similartype to a common representation.

(2) The apparatus of (1) wherein the common representation is a motionmap.

(3) The apparatus of anyone of (1) or (2) wherein data association isgiven by the assignment of data to cells of the motion map.

(4) The apparatus of anyone of (1) to (3) wherein the circuitry isconfigured to fuse the data of a Doppler radar and a stereo camera intoa common representation.

(5) The apparatus of anyone of (1) to (4) wherein the circuitry isconfigured to transfer to the common representation motion informationobtained from a Doppler radar.

(6) The apparatus of (4) or (5) wherein the motion information obtainedfrom the Doppler radar comprises polar coordinates of a cell in thepolar coordinate space and angular and radial components of the velocityattributed to the cell.

(7) The apparatus of anyone of (1) to (6) wherein the circuitry isconfigured to transfer to the common representation motion informationobtained by scene flow estimation on images captured by a stereo camera.

(8) The apparatus of (4) or (7) wherein the motion information obtainedfrom scene flow estimation comprises image positions, disparity data anddisplacement data.

(9) The apparatus of anyone of (1) to (6) wherein the circuitry isconfigured to reconstruct position and velocity from sensor data toobtain a motion map of the sensor data.

(10) The apparatus of anyone of (1) to (9) wherein the circuitry isconfigured to apply an error model for motion on the processed data toobtain a motion map of the sensor data.

(11) The apparatus of anyone of (1) to (10) wherein the circuitry isconfigured to represent the motion map represented by μ_(v) _(x) _(,v)_(z) , Σ_(v) _(x) _(,v) _(z) ; x, z where x, z are 2d Cartesiancoordinates in the motion map, μ_(v) _(x) _(,v) _(z) are the mean valuesof a normal distribution describing a velocity estimate (v_(x), v_(z)),and Σ_(v) _(x) _(,v) _(z) is the covariance matrix that represents theinformation on the motion measurement obtained from a sensor-specificerror model.

(12) The apparatus of anyone of (1) to (11) wherein the circuitry isconfigured to fuse sensor data cell-wise by determining for each cell ajoint probability for the velocity.

(13) The apparatus of anyone of (1) to (12) wherein the circuitry isconfigured to assume, for every grid cell of a motion map, that thevelocity v of contacts in this cell follow a normal distribution.

(14) The apparatus of anyone of (1) to (13) wherein the circuitry isconfigured to transform parameters of an error model into an auxiliaryrepresentation P, b according to:P=Σ ⁻¹ andb=Σ ⁻¹ μ=Pμwhere μ is a mean value and Σ is a covariance matrix.

(15) The apparatus of anyone of (1) to (14) wherein the circuitry isconfigured to fuse information from sensors “a” and “b” asp(v|a,b)=N(μ_(c),Σ_(c))∝p(v|a)p(v|b)=N(μ_(a),Σ_(a))N(μ_(b),Σ_(b))withp(v|a)=N(μ_(a),Σ_(a)) and p(v|b)=N(μ_(b),Σ_(b))being two given distributions for v in the same cell and where μ_(a),μ_(b) are the respective mean values of the normal distribution N andΣ_(a), Σ_(b) are the respective covariance matrices of the normaldistribution N.

(16) Vehicle control system comprising the apparatus of anyone of (1) to(15).

(17) Vehicle control system of (16) wherein the vehicle is a motorvehicle, an electric vehicle, a hybrid vehicle, a robot, an autonomousrobot, a drone product, or an autonomous drone product.

(18) Advanced driver assistance system comprising the apparatus ofanyone of (1) to (15).

(19) A method comprising transferring motion information obtained from aplurality of sensors of different or similar type to a commonrepresentation.

(20) A computer program comprising instructions which when carried outon a processor cause the processor to transfer motion informationobtained from a plurality of sensors of different or similar type to acommon representation.

(21) A tangible and readable storage medium storing a computer programcomprising instructions which when carried out on a processor cause theprocessor to transfer motion information obtained from a plurality ofsensors of different or similar type to a common representation.

The invention claimed is:
 1. An apparatus comprising circuitryconfigured to: receive motion information regarding a scene obtainedfrom a plurality of motion sensors, the plurality of motion sensorsincluding a first n notion sensor and a second motion sensor, the firstand second motion sensors being at least two of a first type of motionsensor or at least one of the first type of motion sensor and at leastone of a second type of motion sensor, the second type of motion sensordetecting a different parameter of the scene than the first type ofmotion sensor; transform parameters of an error model into an auxiliaryrepresentation P, b according to:P=Σ ⁻¹ andb=Σ ⁻¹ μ=Pμ where μ as a mean value and Σ is a covariance matrix; applythe error model to data from the first motion sensor to obtain a firstmotion map and to data from the second motion sensor to obtain a secondmotion map; and fuse the first motion map and the second motion map intoa fused motion map.
 2. The apparatus of claim 1, wherein the circuitryis further configured to align data of the first motion sensor and aligndata of the second motion sensor and assign aligned data to cells of thefused motion map.
 3. The apparatus of claim 1 wherein the the firstmotion sensor is a Doppler radar and the second motion sensor is astereo camera.
 4. The apparatus of claim 3, wherein motion informationobtained from the Doppler radar comprises polar coordinates of a cell inthe polar coordinate space and angular and radial components of thevelocity attributed to the cell.
 5. The apparatus of claim 1, whereinthe at least one of the first and second motion sensors is a Dopplerradar.
 6. The apparatus of claim 1, wherein at least one of the firstand second motion sensors is a stereo camera and the circuitry isconfigured to obtain scene flow estimation on images captured by thestereo camera.
 7. The apparatus of claim 6, wherein motion informationobtained from scene flow estimation comprises image positions, disparitydata and displacement data.
 8. The apparatus of claim 1, wherein thecircuitry is configured to reconstruct position and velocity from sensordata to obtain first and second motion maps.
 9. The apparatus of claim 1wherein the circuitry is configured to represent the motion maprepresented by μ_(v) _(x) _(,v) _(z) , Σ_(v) _(x) _(,v) _(z) ; x, z are2d Cartesian coordinates in the motion map, are the mean values of anormal distribution describing a velocity estimate (v_(x), v_(z)), andΣ_(v) _(x) _(, v) _(z) is the covariance matrix that represents theinformation on the motion measurement obtained from a sensor-specificerror model.
 10. The apparatus of claim 1, wherein the circuitry isconfigured to fuse first and second motion mans cell-wise by determiningfor each cell a joint probability for the velocity.
 11. The apparatus ofclaim 1 wherein the circuitry is configured to assume, for every gridcell of the first and second motion maps, that the velocity v ofcontacts in this cell follow a normal distribution.
 12. The apparatus ofclaim 1, wherein the circuitry is configured to fuse information fromthe first motion map of the first motion sensor “a” and the secondmotion map of the second motion sensor “b” asp(v|a,b)=N(μ_(c),Σ_(c))∝p(v|a)p(v|b)=N(μ_(a),Σ_(a))N(μ_(b),Σ_(b))withp(v|a)=N(μ_(a),Σ_(a)) and p(v|b)=N(μ_(b),Σ_(b)) being two givendistributions for v in the same cell and where μ_(a), μ_(b) are therespective mean values of the normal distribution and Σ_(a), Σ_(b) arethe respective covariance matrices of the normal distribution N. 13.Vehicle control system comprising the apparatus of claim
 1. 14. Vehiclecontrol system of claim 13 wherein the vehicle is a motor vehicle, anelectric vehicle, a hybrid vehicle, a robot, an autonomous robot, adrone product, or an autonomous drone product.
 15. Advanced driverassistance system comprising the apparatus of claim
 1. 16. A methodcomprising: receiving motion information regarding a scene obtained froma plurality of motion sensors, the plurality of motion sensors includinga first motion sensor and a second motion sensor, the first and secondmotion sensors being at least two of a first type of motion sensor or atleast one of the first type of motion sensor and at least one of asecond type of motion sensor, the second type of motion sensor detectinga different parameter of the scene than the first type of motion sensor;and fusing information from the first motion sensor “a” and the secondmotion sensor “b” asp(v|a,b)=N(μ_(c),Σ_(c))∝p(v|a)p(v|b)=N(μ_(a),Σ_(a))N(μ_(b),Σ_(b))withp(v|a)=N(μ_(a),Σ_(a)) and p(v|b)=N(μ_(b),Σ_(b)) being two givendistributions for v in the same cell and where μ_(a), μ_(b) are therespective mean values of the normal distribution and Σ_(a), Σ_(b) arethe respective covariance matrices of the normal distribution N into afused motion map.
 17. A non-transitory computer readable storage devicehaving computer readable instructions which, when carried out on aprocessor, cause the processor to: receive motion information regardinga scene obtained from a plurality of motion sensors, the plurality ofmotion sensors including a first motion sensor and a second motionsensor, the first and second motion sensors being at least two of afirst type of motion sensor or at least one of the first type of motionsensor and at least one of a second type of motion sensor, the secondtype of motion sensor detecting a different parameter of the scene thanthe first type of motion sensor; and fuse information from the firstemotion sensor “a” and the second motion sensor “b” asp(v|a,b)=N(μ_(c),Σ_(c))∝p(v|a)p(v|b)=N(μ_(a),Σ_(a))N(μ_(b),Σ_(b))withp(v|a)=N(μ_(a),Σ_(a)) and p(v|b)=N(μ_(b),Σ_(b)) being two givendistributions for v in the same cell and where μ_(a), μ_(b) are therespective mean values of the normal distribution and Σ_(a), Σ_(b) arethe respective covariance matrices of the normal distribution N into afused motion map.
 18. Vehicle control system comprising thenon-transitory computer readable storage device of claim 17.